Operator control unit with tracking

ABSTRACT

An apparatus equipped with an electronic camera, lensed optics, and a visual display in communication with the optics. An analog or digital video signal is conveyed to an operator of the apparatus through the visual display. The apparatus includes an embedded processor to track the orientation and position of the apparatus. Orientation and position information of the apparatus is used to dynamically recalculate display information. In addition, the apparatus may be in communication with a remote device having digital camera optics. Orientation and position information of the apparatus may be conveyed to the remote device to alter the orientation and position of the associated electronic camera optics. The global position and orientation data is refined through the use of a machine vision algorithm. Accordingly, data conveyed to the operator of the apparatus is in relation to the orientation and position of the apparatus and/or the associated orientation and position of the remote device.

CROSS REFERENCE TO RELATED APPLICATION(S)

The present application is a continuation-in-part utility applicationclaiming the benefit of U.S. patent application Ser. No. 10/739,603,filed on Dec. 18, 2003, and titled “Operator Control Unit withTracking,” now pending, which is hereby incorporated by reference.

BACKGROUND OF THE INVENTION

1. Technical Field

This invention relates to an apparatus for remote communication. Morespecifically, the apparatus is adapted to convey information pertainingto the operator with respect to the locale of the apparatus and/or aremote device in communication with the apparatus.

2. Description of the Prior Art

Portable computing apparatus, such as laptop computers and personaldigital apparatus are commonly used for remote computing needs andcommunication with computer systems and networks. A person utilizingsuch apparatus can enter data into the apparatus as long as theapparatus has an input device and source of power.

Many known portable computing apparatus contain communicationelectronics, such as a modem, to enable the operator to send and receivedata to and from the apparatus and other computer systems or networks.Most modems require the operator to physically connect their apparatusto a telecommunication link. However, recently developments forcommunication apparatus capable of transmitting and receiving data froma remote device through a wireless connection include radio frequencytransceivers. Accordingly, portable computing apparatus, which enableoperators to remotely communicate with other devices and transmit datato and receive data from other devices, is common in the art.

There are several apparatus that enable remote communication. Forexample, laptop computers enable people to do computing from arelatively compact personal computer and transmit data through aconnection to a network or other computer system. Similarly, personaldigital apparatus with communications hardware enable users to do remotecomputing on a more limited basis and to transmit files to remote devicethrough a communications connection to a computer network. However,neither the laptop nor the personal digital apparatus is designed toaccount for the physical environment of the unit in which the embeddedprocessor is housed, and to communication the physical environment tothe operator. In addition, laptops, personal digital apparatus, andsimilar computing apparatus are not generally designed to enablewireless communication with another remote device other than computerapparatus or enable bidirectional communication with such apparatus.Accordingly, what is desired is an embedded processor, which can be wornon a body part of the user, that enables remote wireless communicationwith a remote device while accounting for the physical environment andpositioning of the processor.

SUMMARY OF THE INVENTION

This invention comprises a control unit for remote communication.

In one aspect of the invention, an operator control apparatus isprovided with digital camera optics in communication with a visualdisplay, with the optics configured to provide a digital video signal.An embedded processor is provided in communication with the optics. Theprocessor tracks change to orientation and position of the apparatus andrecalculates data to be displayed based on the change. A remote deviceis provided in communication with and separate from the apparatus. Theremote device has an actuator that is configured to be controlled by aninput device of the apparatus. The remote device communicates globalposition data of an object of interest to the processor, the data isthen refined by one or more machine vision algorithms. The processor isconfigured to re-calculate location of the object of interest relativeto the apparatus. In addition, the visual display employs an overlay toshow a combination of data received from the optics local to the visualdisplay and the remote device. This overlay provides position andorientation in three dimensional spaces to a location of the objectrelative to location and orientation of the apparatus.

In another aspect of the invention, a method is provided for remotecommunication. A digital video feed is provided to a visual displaythrough optics. Orientation and position change of an apparatus incommunication with said visual display are tracked. The orientation of aportion of a device remote from said apparatus is controlled throughorientation of the apparatus. The remote device includes a globalpositioning sensor. Both global position and orientation data of anobject of interest are communicated to the visual display. Thiscommunication includes refining the global position data with one ormore machine vision algorithms. The visual display shows a combinationof data received from the optics and the remote device, with thecombination providing location data of an object relative to location ofthe apparatus.

In yet another aspect of the invention, an article is provided withoptics in communication with a visual display. The optics are configuredto provide a digital video signal. The article includes acomputer-readable medium encoded with instructions to be executed by acomputer. Instructions are provided to provide a digital video feed to avisual display through optics local to the visual display and remotefrom the visual display, and to track orientation and position of anapparatus in communication with the visual display. In addition,instructions are provided to control orientation of a portion of adevice remote from the apparatus through orientation of the apparatus.Instructions are also provided to communicate global position of anobject of interest as detected from the remote device to the visualdisplay, including instructions to refine the global position andorientation data by employment of a machine vision algorithm. Finally,instructions are provided to present a combination of data collected bythe local optics and the remote device, this combination providinglocation of an object relative to location of the apparatus.

Other features and advantages of this invention will become apparentfrom the following detailed description of the presently preferredembodiment of the invention, taken-in conjunction with the accompanyingdrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings referenced herein form a part of the specification.Features shown in the drawings are meant as illustrative of only someembodiments of the invention, and not of all embodiments of theinvention unless otherwise explicitly indicated.

FIG. 1 is a perspective view of the operator control unit according tothe preferred embodiment of this invention, and is suggested forprinting on the first page of the issued patent.

FIG. 2 is a flow diagram illustrating the local situational awarenessmode.

FIG. 3 is an illustration of a graphical user interface with dataoverlay.

FIG. 4 is a flow diagram illustrating the remote situational awarenessmode.

FIG. 5 is a flow diagram illustrating the birds eye map mode.

FIG. 6 is a flow diagram illustrating the first person map mode.

FIG. 7 is a perspective view of the operator control unit with atethered computation device.

FIG. 8 is a block diagram of a set of tools to support remotecommunication between an operator control apparatus and a remote object.

DESCRIPTION OF THE PREFERRED EMBODIMENT

It will be readily understood that the components of the presentinvention, as generally described and illustrated in the Figures herein,may be arranged and designed in a wide variety of differentconfigurations. Thus, the following detailed description of theembodiments of the apparatus, system, and method of the presentinvention, as presented in the Figures, is not intended to limit thescope of the invention, as claimed, but is merely representative ofselected embodiments of the invention.

Reference throughout this specification to “a select embodiment,” “oneembodiment,” or “an embodiment” means that a particular feature,structure, or characteristic described in connection with the embodimentis included in at least one embodiment of the present invention. Thus,appearances of the phrases “a select embodiment,” “in one embodiment,”or “in an embodiment” in various places throughout this specificationare not necessarily referring to the same embodiment.

Furthermore, the described features, structures, or characteristics maybe combined in any suitable manner in one or more embodiments. In thefollowing description, numerous specific details are provided, such asexamples of vision and vision techniques, to provide a thoroughunderstanding of embodiments of the invention. One skilled in therelevant art will recognize, however, that the invention can bepracticed without one or more of the specific details, or with othermethods, components, materials, etc. In other instances, well-knownstructures, materials, or operations are not shown or described indetail to avoid obscuring aspects of the invention.

The illustrated embodiments of the invention will be best understood byreference to the drawings, wherein like parts are designated by likenumerals throughout. The following description is intended only by wayof example, and simply illustrates certain selected embodiments ofdevices, systems, and processes that are consistent with the inventionas claimed herein.

Overview

An apparatus for conveying local and/or remote information to anoperator is provided. The positioning of the apparatus may control theinformation conveyed to the operator. An embedded processor of a controlunit computes the position and orientation of the apparatus and gathersdata associated therewith. One or more machine vision algorithms areemployed to accurately refine the position and orientation calculationsof the apparatus. Machine vision algorithms are used to recognizeobjects in three dimensional spaces. In addition, the apparatus maycommunicate with a remote device. The orientation of the apparatus maybe used to control the orientation of the remote device, and associateddata gathered from the remote device and transmitted to the apparatus.Accordingly, the position and orientation of the apparatus control thedata gathered and conveyed to the operator.

Technical Details

As shown in FIG. 1, the control unit 10 is in the physical formrepresentative of a binocular. The control unit 10 may be hand held, orworn around a body part of the operator with a strap 5. The control unit10 has a case 12 adapted to house internal components, such as sensorsand I/O apparatus. Data processing is performed by a computation device20 that is shown embedded to the control unit 10. However, in analternative embodiment, as shown in FIG. 7, the computation unit 20 maybe tethered to the control unit 10 by a signal and power cable 22. Thecomputation unit 20 includes a computer with an embedded processor.Machine vision algorithms may run on the embedded processor of thecomputation unit 20 or on a separate processor. More specifically, themachine vision algorithm(s) will count, measure, and/or identifyobjects, dimensions, or other features in the image. In one embodiment,an edge and vertex detection algorithm identifies points in a digitalimage at which the image brightness changes, e.g. has discontinuities.The purpose of detecting sharp changes in image brightness is to captureimportant events and changes in properties. The result of applying anedge detector to an image is a set of connected curves that indicatesthe boundaries of the object(s), the boundaries of surface markings, aswell as curves that correspond to discontinuities in surfaceorientation. Accordingly, application of an edge and vertex detection toan image reduces the amount of data to be processed and filters outinformation that may be regarded as less relevant, while preserving theimportant structural or environmental properties of an image in order tomatch these structural or environmental properties to pre-stored data.

Preferably, the embedded processor includes a wireless communicationapparatus to enable communication between the embedded processor and aremote device. The case 12 has a proximal end 14 and a distal end 16. Aset of ear pieces 32, 36 are mounted adjacent to the proximal end 14 forreceipt of auditory data. External sound sources are damped by pliablematerial 34, 38 on the earpieces 32, 36, respectively, resulting inenhanced clarity of presentation of the auditory data to the operator.The control unit 10 has a directional microphone 40 to detect auditorydata conveyed to the earpiece. Similarly, a set of eyepieces 42, 46 aremounted adjacent to the proximal end 14 for receipt and presentation ofvisual data to the operator. External light sources are shielded fromthe display using pliable material 44, 48 that conforms to theoperator's face. Within the pliable material 44, 48 of eyepieces 42, 46are pressure sensors (not shown) indicating proximity of the operatorsface with respect to the control unit. Both the ear and eye pieces areadapted to receive data in stereo format. In addition, the control unit10 includes a light sensor 50, a light amplification sensor array 52,digital video camera optics (not shown), an infra-red amplificationsensor array 54 to convey visual data to the operator through theeyepieces 42, 46, and lens optics 82 and 84 to provide a magnifiedanalog display to the operator. Accordingly, the control unit 10includes apparatus for conveying auditory and visual information to anoperator of the unit.

In addition to conveying information to the operator of the unit, inputapparatus are provided to collect data as well as enable communicationbetween the operator and the unit, and/or between the operator and aremote device. A set of input devices 60 and 70 are provided on eachlateral side of the control unit 10. The input devices preferablyinclude additional input devices 62, 64, and 66, and 72, 74, and 76,shown in the form of tactile pushbuttons. Each of the input devices ismapped to a set of corresponding logical states in the control unitand/or a remote device. A logical state may correspond to activation ofone or more actuators on the remote device. One or more of the inputdevices may be in the form of a proportional input device, such as aproportional input grip, as shown in FIG. 1. Each proportional inputgrip is preferably enclosed within neoprene boots (not shown) to protectthe components of the proportional input grip from dust and moisture.Other materials may be used to insulate the proportional input gripsfrom dust, moisture, electromagnetic interferences, and any othercondition that would affect communication and operation of theproportional input grip. In addition, the boots function as a sealbetween the input device and the control unit case 12.

Each proportional input grip 60, 70 have a proximal end 61, 71 and adistal end 69, 79, respectively. The distal ends of the proportionalinput grips extend from a surface of the case 12 and may be actuated bythe operator. Similarly, the proximal ends 61, 71 of the proportionalinput grips 60, 70 are connected to electronic circuits that residewithin an interior section of the case 12. As the proportional inputgrip is revolved around its center axis, a signal is produced thatcorresponds to the degree of actuation. The signal is preferably in theform of a voltage output that preferably ranges from 0 to 5 volts, butmay be calibrated for a lesser or greater output. As the proportionalinput grip 60, 70 is rotated about its axis, a proportional voltage isoutput to the associated electronic circuit. Alternatively, theproportional input grip may use optical motion detection, wherein anoptical signal would be digitized at an analog to digital converterbypassing any electronic circuits. Actuation of the proportional inputgrip 60, 70 may be communicated to a respective logical state or motorof the remote device controlling direction, velocity and/or illuminationfor any apparatus adapted to receive the variable input. The signal fromthe circuit board associated with the proportional input device 60, 70is processed by an analog to digital converter to digitize the data intoa computer readable format. Following the digitizing process, theprocessed data is streamed to a communication port of the embeddedprocessor. The radial proportional input grip motion described for theproportional input devices 60, 70 may be replaced by any otherproportional movement that would be necessary to control the remotedevice. However, actuation of the proportional input grip is not limitedto communication with a remote device. The proportional input grip mayalso be used to communicate with the visual display. Accordingly, theproportional input device functions as an input device in communicationwith the control unit 10 to provide a proportional signal to theembedded processor of the control unit and/or a remote device.

As with the proportional input devices 60, 70, the tactile buttons 62,64, 66, 72, 74, 76 convey information from the operator to a circuitboard associated therewith, which transmits the data to ananalog-digital converter. Wired communication electronics are integratedinto the analog-digital converter to digitize the data into a computerreadable format and to communicate data received from the input deviceto the embedded processor or streamed to a communication port of theembedded processor. The tactile buttons may be used to communicate witheither the visual display or the remote device, or both. Functionalityassociated with the tactile pushbuttons may include, switching modes ofoperation, switching proximity sensors, and navigation within agraphical user interface. Pressure sensors in the proportional inputdevice, known in the art as “dead man” switches, control communicationsignals between the control unit 10 and the remote device. For example,a release of one of the pressure sensors sends a communication signal tothe remote device to enter a safe state. Whereas, when the pressuresensor is engaged, communication between the control unit 10 and theremote device can be achieved. In a preferred embodiment, the tactilepushbuttons are separated by a silicone rubber membrane to preventmoisture and dust from entering the case 12. However, the membrane maybe comprised of an alternative material that provides protection of theinterior section of the case and associated circuit board(s) from damagedue to dust, moisture, and environmental weather conditions.Accordingly, actuation of the tactile pushbuttons enables an operator ofthe unit to communicate a variety of signals to the embedded processorfor local or remote communication.

The hardware components of the control unit 10 may be used to visuallyconvey data from a remote device to an operator of the unit 10. Visualdata are displayed to the operator on the visual display as seen throughthe eyepieces 42 and 46. There are four modes of operation for visualdisplay, including a local situational awareness (LSAM), remotesituational awareness (RSAM), first person map (FPMM), and bird's eyemap (BEMM). The control unit 10 includes several apparatus to operate ineach of these modes. For example, a global positioning system (GPS)sensor (not shown) is provided to convey the location of the controlunit 10 to the embedded processor of the control unit. An electroniccompass (not shown) and an electronic accelerometer (not shown) areprovided to convey direction with respect to North and angle withrespect to the Horizon, respectively, to the embedded processor of thecontrol unit 10. Similarly, all position and orientation informationgathered by the remote device are conveyed to the embedded processor ofthe control unit. In addition, a rangefinder 56 is provided both on thecontrol unit 10 and the remote device. The rangefinder conveys distanceto a specific object or location by calculating a range to objects ofinterest. In one embodiment, the rangefinder may be in the form of anelectromagnetic signal. Accordingly, the apparatus of the control unitincludes tools to collect appropriate data to enable the four modes ofoperation.

FIG. 2 is a flow diagram 100 illustrating process of conveying data toan operator utilizing the local situational awareness mode (LSAM) of thecontrol unit 10. When the control unit 10 is operated in the localsituational awareness mode (LSAM), the operator can enhance his/hervision of immediate surroundings through video data from the lightamplification sensor array 52, lens optics 82 and 84, or both. The firststep in entering the local situational awareness mode is for theembedded processor of the control unit 10 to receive global positiondata from the GPS sensor of the control unit 102. Thereafter, theembedded processor of the control unit 10 receives global orientationdata from the electronic accelerometer and electronic compass of thecontrol unit 104. At step 106, one or more machine vision algorithms areexecuted to refine and increase the accuracy of calculations of positionand orientation of data received in steps 102 and 104. In oneembodiment, the machine vision algorithms allow one to recognize one ormore objects within an image provided by optics. Following step 106, adatabase containing information as well as position data is searched forobjects located within a local line of sight. In one embodiment, thedatabase is a geographical database. The list of objects is receivedfrom the database at step 108, after which the object(s) recognized bythe machine vision algorithms at step 106 are matched to one or moreobjects received from the database 110. Thereafter, the absoluteposition and orientation estimations for the control unit are refined atsteps 112 and 114, respectively, using absolute position data for theobject(s) obtained at step 110 from the database.

Following step 114, object data update is received 116. The location ofthe object(s) is re-calculated relative to the control unit 118.Thereafter, infra-red sensor array data is collected and received 120,and the location of the infra-red sources are calculated relative to thelocation of the control unit 122. Information gathered by the remotedevice or any other source(s) relative to the object(s) is displayed ina transparent overlay form relative to the actual position of theobject(s) with respect to the position and orientation of the controlunit 124. Such information may include infra-red source data.Accordingly, the local situational awareness mode (LSAM), together withthe machine vision algorithm, calculates position and orientation inthree dimensional spaces to accurately represent graphical overlays ofenvironmental information to an operator of the control unit.

The overlay information gathered can indicate the location of objectswhich are not directly visible to the operator. In addition, the overlayinformation provides information about objects which are visible to theoperator. The execution of machine vision algorithms, as discussedabove, provides accurate refinement of the absolute position of objectsin three dimensions as well as accurate representation of graphicaloverlays of environmental information to the operator. FIG. 3 is apanoramic view 120 of a visual display in the local situationalawareness mode (LSAM). There are two noted objects, object 125 which isnot directly visible to the operator, and object 130 which is visible tothe operator. The distance of the objects 125 a and 130 a to the controlunit are noted adjacent to each object. In this example, the objects are200 meters and 27 meters, respectively. Infra-red sensor data 132 isdisplayed relative to the actual location of the infra-red source. Thedata overlay display may optionally include telemetry data from theremote device 148 as transparent text 134 and/or graphics display 136.Global orientation data 138 and position information 140 may also beprovided in the display. In addition, standard map symbols representingconventional objects are represented, as well as grid lines 144 and 146,representing topographical information. For example, a railway line 142is shown. Accordingly, in the local situational awareness mode (LSAM),an operator of the control unit may enhance his/her vision of his/hersurroundings through video data from the light amplification sensorarray of the control unit and/or through lens optics of the controlunit.

FIG. 4 is a flow diagram 160 illustrating a process of conveying data toan operator utilizing the remote situational awareness mode (RSAM) ofthe control unit. When the control unit 10 is operated in the remotesituational awareness mode (RSAM), the operator requests a change inorientation of a camera in communication with the remote device. Thecamera gathers data and communicates that data to the control unit. Inthe remote situational awareness mode (RSAM), a change in theorientation of the control unit corresponds to new orientation data forthe camera of the remote device. The first step in entering the remotesituational awareness mode (RSAM) is to calculate the orientation of thecontrol unit 162. Thereafter, any change in orientation from the priorposition data of the control unit is calculated 164. The change in theorientation of the control unit is transmitted to the remote device 166.Following transmission of the orientation change, the remote devicemodifies the orientation and/or position of it's camera to reflect thechanges communicated from the control unit 168. Thereafter, the controlunit receives a video signal from the remote device 170, and displaysthe video signal to the operator 172. The purpose of the remotesituational awareness mode (RSAM) is to convey a change in thepositioning of the remote device and associated camera. The orientationof the control unit 10 directly controls the orientation of the videosensors on the remote device. The combination of sending orientationchanges and receiving video signal(s) is a form of bi-directionalcommunication between the control unit and the remote device. Thebi-directional communication between the control unit and the remotedevice is interactive by its nature. The orientation and position of thevideo sensor on the remote device are mapped to coincide with theorientation and position of the control unit 10. Accordingly, the neworientation of the camera of the remote device enables the remote deviceto transmit data from a new orientation and to focus on changes in oneor more objects or on one or more new objects.

FIG. 5 is a flow diagram 180 illustrating the process of conveying mapdata to an operator utilizing the birds eye map mode (BEMM). The purposeof this mode is to provide three dimensional map data to the controlunit visible to the operator through the visual display. Followinginitiation of the birds eye map mode, the embedded processor of thecontrol unit 10 receives global position data from the GPS sensor of thecontrol unit 182. Thereafter, the embedded processor of the control unit10 receives global orientation data from the electronic accelerometerand electronic compass of the control unit 184. Upon receiving the dataat steps 182 and 184, the processor calculates position and orientationof the control unit 186. Following receipt and calculation of controlunit position data, object of interest data is received 188. In oneembodiment, object data pertains to features of terrain and/orenvironment, and objects of interest data include tactical data as asubset of object data. The location of the object(s) of interest iscalculated relative to the control unit 190. Map data is retrieved froma data storage medium in communication with the embedded processor ofthe control unit 191. Thereafter, a new three dimensional map is createdand sent to the visual display of the control unit for use by theoperator 192. Information gathered by the remote device or any othersource(s) relative to the object of interest is displayed in an overlayform relative to the actual position of the object(s) of interest withrespect to the position and orientation of the control unit 194. In theBEMM, the control unit 10 displays three dimensional map data to theoperator as if the operator were a set distance above his/her currentposition, or that of the position of the remote device, i.e. lookingdown. The map information is displayed with proper orientation to northtogether with the current location of the control unit 10 and the remotedevice. In this mode, as the operator orients and changes the controlunit 10, the map data changes accordingly. Preferably, terrain detail isdisplayed as a wireframe, and natural and artificial objects aredisplayed using standardized coded map symbols. Map data is stored inpersistent memory and may be updated by satellite data and remote pilotvehicles. Accordingly, the birds eye map mode (BEMM) is intended toretrieve and convey map data based upon orientation of the control unit.

FIG. 6 is a flow diagram 200 illustrating the process of obtaininglocation for objects of interest in a first person map mode (FPMM).Global position data is obtained from a GPS sensor associated with theremote device 202. Thereafter global orientation data is obtained froman electronic compass associated with the control unit 204. The positionand orientation of the remote device is recalculated from a priorcalculation based upon readings obtained from the associated GPS sensor,electronic accelerometer, and electronic compass 206. Similarly, dataassociated with any objects of interest must be obtained 208.Thereafter, the location of the objects of interest is re-calculatedbased upon any new position data obtained from the remote device 210.Following step 210, infra-red sensor array data is collected 212 andcalculated relative to the position of the infra-red sources 214. Onceall of the data from the remote device and objects of interest areobtained, three dimensional graph data for a specific orientation andposition is calculated 216. Map data is retrieved from a data storagemedium in communication with the embedded processor of the control unit218. Thereafter, a map is made visible to the operator of the controlunit through the visual display 220. The map is preferably a threedimensional map with data projected as transparent overlay graphics. Theproject data includes infra-red source data, objects of interest, globalposition and orientation data, map data, and remote device data.

Embodiments within the scope of the present invention also includearticles of manufacture comprising program storage means having encodedtherein program code to communicate data between the input device anddata presented on the visual display. Such program storage means can beany available media which can be accessed by a general purpose orspecial purpose computer. By way of example, and not limitation, suchprogram storage means can include RAM, ROM, EPROM, CD-ROM, or otheroptical disk storage, magnetic disk storage or other magnetic storagedevices, or any other medium which can be used to store the desiredprogram code means and which can be accessed by a general purpose orspecial purpose computer. Combinations of the above should also beincluded in the scope of the program storage means.

FIG. 8 is a block diagram (800) illustrating a set of tools to supportremote communication between an operator control apparatus and a remoteobject. As shown, a computer based system (802) is provided with aprocessor chip (804) coupled to memory (806) by a bus structure (810).Although only one processor chip (804) is shown, in one embodiment, moreprocessor chips may be provided in an expanded design. Data storage(820) is provided in communication with the processor chip (804), andthe data storage (820) is provided with a database (822) of one or moreobjects. In one embodiment, the database (822) is a geographicaldatabase to store geographical information for one or more objects ofinterest. The system (802) is further configured with a set of tools tomanage the remote communication. More specifically, a set of managersare provided to support the functionality of the remote communication.An object manager (830) is provided local to memory (806) and isemployed to match an object of interest to an object contained in adatabase. In addition, a refinement manager (840) is provided local tomemory (806), and is employed to communicate with the object manager(830). The refinement manager (840) is responsible for refining globalposition and orientation data as gathered by the control unit based uponglobal position data stored in the database (822) for an object ofinterest. Both the object manager (830) and the refinement manager (840)are shown local to memory (806). However, the invention should not belimited to this embodiment. For example, in one embodiment, the objectmanager (830) and/or the refinement manager (840) may reside as hardwaretools external to local memory (806), or they may be implemented as acombination of hardware and software. Similarly, in one embodiment, theobject manager (830) and refinement manager (840) may reside on a remotesystem in communication with the processor chip (804). Accordingly, themanagers may be implemented as a software tool or a hardware tool tomanage remote communication of data, and more specifically to support anoverlay of data in three dimensional spaces.

In one embodiment, the invention is implemented in software, whichincludes but is not limited to firmware, resident software, microcode,etc. The invention can take the form of a computer program productaccessible from a computer-usable or computer-readable medium providingprogram code for use by or in connection with a computer or anyinstruction execution system. For the purposes of this description, acomputer-usable or computer readable medium can be any apparatus thatcan contain, store, communicate, propagate, or transport the program foruse by or in connection with the instruction execution system,apparatus, or device.

The medium can be an electronic, magnetic, optical, electromagnetic,infrared, or semiconductor system (or apparatus or device) orpropagation medium. Examples of a computer-readable medium include butare not limited to a semiconductor or solid state memory, magnetic tape,a removable computer diskette, random access memory (RAM), read-onlymemory (ROM), a rigid magnetic disk, and an optical disk. Currentexamples of optical disks include compact disk B read only (CD-ROM),compact disk B read/write (CD-R/W) and DVD.

A data processing system suitable for storing and/or executing programcode includes at least one processor coupled directly or indirectly tomemory elements through a system bus. The memory elements can includelocal memory employed during actual execution of the program code, bulkstorage, and cache memories which provide temporary storage of at leastsome program code in order to reduce the number of times code must beretrieved from bulk storage during execution.

Input/output or I/O devices (including but not limited to keyboards,displays, pointing devices, etc.) can be coupled to the system eitherdirectly or through intervening I/O controllers. Network adapters mayalso be coupled to the system to enable the data processing system tobecome coupled to other data processing systems or remote printers orstorage devices through intervening private or public networks. Thesoftware implementation can take the form of a computer program productaccessible from a computer-useable or computer-readable medium providingprogram code for use by or in connection with a computer or anyinstruction execution system.

It will be appreciated that, although specific embodiments of theinvention have been described herein for purposes of illustration,various modifications may be made without departing from the spirit andscope of the invention. In particular, the input devices may come indifferent forms, including a proportional input device, such as ajoystick, a rocker pad, a touch pad, a track balls, and alternativeinput devices. Additionally, the invention should not be limited to themappings of the input devices to the described movement andcommunication with the image on the visual display. In one embodiment,there may be different mappings of the input devices to the image, oreven additional mappings for different image movements. Furthermore, theinvention should not be limited to a fixed set of mappings. In oneembodiment, an interface may be provided to modify the mappings of theinput devices. Accordingly, the scope of protection of this invention islimited only by the following claims and their equivalents.

Advantages Over the Prior Art

The embedded processor of the control unit tracks orientation andposition of the control unit 10. Positioning of the control apparatus isconveyed to digital camera optics in communication with the embeddedprocessor. Since the control unit 10 is adapted to be placed against theeyes and/or ears of the operator during use, the position andorientation of the control unit 10 is directly related to theorientation and position of the head of the operator of the control unit10. The orientation and position information of the control unit may beprojected onto the visual display of the control unit. In addition, theorientation and position of the control unit 10 may be conveyed to theremote device and the associated digital camera optics to position thecamera associated with the remote device in accordance with theorientation and position of the control unit 10. Communication oforientation and position data enhances interactivity between the controlunit and the remote device, aside from the environment of the remotedevice. In addition, the embedded processor may create a wireframe togive shape to the terrain and synthetic graphics to represent physicalitems in the noted relative locations, thus producing synthetic vision.The use of a wireframe and/or synthetic graphics timely conveys map,terrain, and shape data to the visual display.

Alternative Embodiments

It will be appreciated that, although specific embodiments of theinvention have been described herein for purposes of illustration,various modifications may be made without departing from the spirit andscope of the invention. In particular, the control unit may be designedto communicate with a variety of remote device. For example, the remotedevice may be in an electronic or mechanical form with logical statesmapped to corresponding input devices and motors of the control unit.The remote device may include a camera that captures live video toprovide live video feedback to the control unit. In addition, thecontrol unit may be used to download topographical and/or geographicaldata independent of or in conjunction with the various modes ofoperation. The visual display may be in the form of a liquid crystaldisplay, or an alternative medium that enables viewing by the operatorwhile maintaining the integrity of the control unit. Similarly, thewireless communication electronics may be in the form of wirelesscommunication electronics in communication with the embedded processorof the control unit, or an alternative communication electronics thatenables wireless communication of data between the embedded processorand a corresponding wireless communication apparatus remote from thecontrol unit. In addition, the scope of the invention should not belimited to the input devices described together with the control unit.Alternative input devices that enable communication of data between thecontrol unit and the remote device may be employed. Accordingly, thescope of protection of this invention is limited only by the followingclaims and their equivalents.

1. An operator control apparatus, comprising: optics in communicationwith a visual display, wherein said optics is adapted to provide adigital video signal; an embedded processor in communication with saidoptics adapted to track change to orientation and position of saidapparatus and to re-calculate data to be displayed based on said change;a remote device in communication with said apparatus and separate fromsaid apparatus, said remote device having an actuator adapted to becontrolled by an input device of said apparatus, and said remote deviceto communicate global position data of an object of interest to saidprocessor, said global position data of the object of interestrecalculated with respect to the position of said apparatus; and saidprocessor to re-calculate location of the object of interest relative tosaid apparatus; and said visual display to employ an overlay to show acombination of data received from the optics local to the visual displayand the remote device, said overlay to provide position and orientationin three dimensional space to a location of the object relative tolocation and orientation of said apparatus.
 2. The apparatus of claim 1,further comprising said global position and orientation data is refinedby employment of a machine vision algorithm, said machine visionalgorithm to recognize said object of interest within an image receivedby said optics.
 3. The apparatus of claim 2, further comprising anobject manager to match said object of interest recognized by saidmachine vision algorithm to an object contained in a database.
 4. Theapparatus of claim 3, further comprising global position data stored insaid database, and said object manager to match said global positiondata to a corresponding object of interest recognized by said machinevision algorithm.
 5. The apparatus of claim 4, further comprising arefinement manager in communication with said object manager, saidrefinement manager to refine said global position and orientation databased upon said global position data stored in the database for theobject of interest.
 6. The apparatus of claim 2, further comprising amachine vision algorithm to recognize data selected from the groupconsisting of: features of terrain, environment, objects of interest,and tactical data.
 7. A method for remote communication comprising:providing a digital video feed to a visual display through optics;tracking orientation and position change of an apparatus incommunication with said visual display; controlling orientation of aportion of a device remote from said apparatus through orientation ofsaid apparatus, wherein said remote device includes a global positioningsensor; communicating global position and orientation data of an objectof interest to said visual display, said global position data of theobject of interest recalculated with respect to the position of saidapparatus; and said visual display showing a combination of datareceived from said optics and said remote device, said combinationproviding location data of an object relative to location of saidapparatus.
 8. The method of claim 7, further comprising refining saidglobal position data with a machine vision algorithm and recognizing theobject of interest within an image received by said optics.
 9. Themethod of claim 8, further comprising matching said recognized object ofinterest with an object contained in a database.
 10. The method of claim9, further comprising storing global position data in said database andmatching said global position data to a corresponding object of interestrecognized by said machine vision algorithm.
 11. The method of claim 10,further comprising refining said global position and orientation data ofsaid apparatus based upon said stored global position data for theobject of interest obtained from said database.
 12. An article withoptics in communication with a visual display, wherein said optics isadapted to provide a digital video signal, the article comprising: acomputer-readable medium encoded with instructions to be executed by acomputer, said instructions comprising: instructions to provide adigital video feed to a visual display through optics local to saidvisual display and remote from said visual display; instructions totrack orientation and position of an apparatus in communication withsaid visual display; instructions to control orientation of a portion ofa device remote from said apparatus through orientation of saidapparatus; instructions to communicate global position of an object ofinterest as detected from said remote device to said visual display,including, recalculating said global position data of the object ofinterest with respect to the position of said apparatus; andinstructions to present a combination of data collected by said localoptics and said remote device, said combination providing location of anobject relative to location of said apparatus.
 13. The article of claim12, further comprising instructions for said a machine vision algorithmto refine said global position and orientation data of said apparatus.14. The article of claim 12, further comprising instructions for saidmachine vision algorithm to recognize said object within an imagereceived by said optics.
 15. The article of claim 14, further comprisinginstructions to match said object recognized by said machine visionalgorithm to an object contained in a database.
 16. The article of claim15, further comprising global position data pre-stored in saidgeographical database, and instructions to match the global positiondata to a corresponding object recognized by said machine visionalgorithm.
 17. The article of claim 16, further comprising instructionsto refine said global position and orientation data of said apparatusbased upon said pre-stored global position data for the object obtainedfrom said database.
 18. An operator control apparatus, comprising:optics in communication with a visual display, wherein said optics isadapted to provide a digital video signal; an embedded processor incommunication with said optics adapted to track change to orientationand position of said apparatus and to recalculate data to be displayedbased on said change; a remote device in communication with saidapparatus and separate from said apparatus, the remote device inelectronic form with logical states mapped to the apparatus, said remotedevice having an actuator adapted to be controlled by an input device ofsaid apparatus, and said remote device to communicate global positiondata of objects of interest to said visual display; said visual displayto show a combination of data received from the optics and the remotedevice; and synthetic vision for data with respect to an object, whereinsaid synthetic vision provides shape to a physical item relative tolocation of said physical item with respect to said apparatus.
 19. Theoperator control apparatus of claim 18, further comprising said logicalstates mapped to corresponding input devices of the apparatus.